1. Field of the Invention
The present invention relates to an image pickup apparatus and an image pickup method for acquiring and combining image data of a plurality of frames.
2. Description of the Related Art
In recent years, as an image pickup apparatus such as a digital camera, there has been proposed an image pickup apparatus that combines images of a plurality of frames continuously read out from an image sensor to thereby realize an image having a new image processing effect that cannot be represented by a photographed image of one frame or acquire an image that cannot be easily obtained by photographing of one frame. The image pickup apparatus has been sold in a market.
For example, Japanese Patent Application Laid-Open Publication No. 2010-136058 (Patent Literature 1) proposes an image pickup apparatus that easily obtains a panning-photographed image by combining images of a plurality of frames.
When panning photographing is performed with a conventional camera, it is not easy to photograph a panning-photographed image as intended by a photographer. More specifically, the photographer sometimes does not know shutter speed suitable for panning photographing that makes a panning amount of a background moderate. Further, the shutter speed suitable for the panning photographing is shutter speed set rather long to allow the background to flow. Therefore, if photographing is performed by setting only the shutter speed suitable for the panning photographing without appropriately setting a diaphragm value and ISO sensitivity, in some case, an exposure amount is too large and excessive exposure occurs.
On the other hand, the image pickup apparatus described in Patent Literature 1 combines images of a plurality of frames consecutively photographed with appropriate exposure to obtain a panning-photographed image. Therefore, it is possible to obtain not only a combined image with appropriate exposure but also a combined image photographed at shutter speed that can show a panning photographing effect desired by the photographer. Further, in the technique described in Patent Literature 1, an object (a background) desired to be panned and an object (a main object) not desired to be panned are detected out of a plurality of objects in image data. Images are combined such that the main object is not panned and the background is panned to make it easier to perform the panning photographing.
When consecutively-photographed images are combined to generate an image of one frame as described in Patent Literature 1, as a consecutive photographing method, it is conceivable to adopt a method of mechanical shutter consecutive photographing for repeatedly performing opening and closing actions of a mechanical shutter and a method of consecutive photographing by a so-called electronic shutter for repeatedly performing readout from an image sensor in a state in which a mechanical shutter is kept opened (hereinafter referred to as electronic readout consecutive photographing as appropriate) like moving image photographing of a general CMOS image sensor.
A panning-photographed image generated by combining images of a plurality of frames photographed by the mechanical shutter consecutive photographing or the electronic readout consecutive photographing is an image different from a panning-photographed image photographed in one frame by a professional photographer, who is familiar with panning photographing, using the mechanical shutter.
More specifically, when the consecutive photographing is performed by the mechanical shutter, a shutter is once closed after an image of one frame is photographed and, thereafter, an action for opening the shutter is performed to photograph an image of the next frame. Therefore, exposure omission in which photographing is not performed occurs among continuous frames because of a time period (e.g., approximately 1/10 (sec)) from the closing to the opening of the shutter. For example, photographing is generally performed at shutter speed of approximately 1/30 to ½ (sec) when motor sports, flying of wild birds, and the like are panning-photographed. Therefore, exposure omission of approximately 1/10 (sec) in order close to the shutter speed occurs among respective images in the mechanical shutter consecutive photographing. When images of a plurality of frames obtained by performing the consecutive photographing are combined to generate a panning-photographed image, there is no sense of discomfort in a main object because the main object stands still. However, a panning state of a background portion becomes discontinuous because of the exposure omission. Therefore, in combining the images to generate the panning-photographed image using the mechanical shutter consecutive photographing, an image in a natural panning state like a panning-photographed image photographed as an image of one frame by a professional photographer cannot be obtained.
On the other hand, the electronic readout consecutive photographing is performed as explained below. In a CMOS image sensor used in a digital camera in recent years, in general, respective signals of a plurality of pixels arranged in row and column directions are sequentially read out for each of rows. At a point in time when readout of accumulated pixel signals is completed for each of the rows, signal accumulation of the pixels in the row is started and readout of the next row is performed. In this way, in the electronic readout consecutive photographing, the consecutive photographing can be performed without interrupting exposure of the respective rows. Therefore, the exposure omission in the case of the mechanical shutter consecutive photographing does not occur. However, a time period from a start of readout of a top row of an image to completion of readout of a bottom row (a so-called traveling time of the electronic shutter) is long compared with a traveling time of the mechanical shutter. As a specific example, whereas the mechanical shutter requires approximately 2 to 4 (msec) to travel from an upper end to a lower end of an image sensor, a time period required for electronic readout of one screen in the general CMOS image sensor is approximately 10 to 100 (msec). Therefore, exposure timing shifts between an upper side portion and a lower side portion of an image when the electronic shutter is used. Consequently, when a moving object is photographed or when a camera itself is moved to photograph an object as in the panning photographing, a phenomenon in which the object is unnaturally distorted occurs in a photographed image.
This is more specifically explained with reference to the drawings. For example, as shown in FIG. 4, it is assumed that a main object MO such as a car traveling from a right side to a left side with respect to a stationary background BO is panning-photographed by directing a camera to the main object MO and swinging the camera from right to left.
At this point, when images of a plurality of frames consecutively photographed by the mechanical shutter are combined, exposure omission occurs among the images of the respective frames as explained above. Therefore, as shown in FIG. 11, an image with an unnatural panning state is obtained in which panning of the background BO is discontinuously interrupted. FIG. 11 is a diagram showing an example of a panning-photographed image SP1 generated by combining the images of the plurality of frames consecutively photographed by the mechanical shutter.
When the mechanical shutter is not used and only the electronic shutter (in an example shown in FIG. 12, an electronic shutter traveling down to up is assumed) is used for all the images of the plurality of frames, a shift of exposure timing on an upper side and a lower side of an image is large concerning all the frames including a first one frame and a last one frame. Therefore, as shown in FIG. 12, a combined image is an unnatural image in which the background BO is distorted more to the right side toward the upper side of the image. FIG. 12 is a diagram showing an example of a panning-photographed image SP2 generated by combining the images of the plurality of frames consecutively photographed by the electronic shutter.
In an image pickup apparatus that combines consecutively-photographed images to generate an image of one frame, the image is an unnatural image compared with an image panning-photographed as an image of one frame because of exposure omission among frames if a method of consecutive photographing is the mechanical shutter consecutive photographing and because of object distortion due to a shift of exposure timing in respective rows if the method of consecutive photographing is the electronic readout consecutive photographing.
The combined image is the unnatural image in this way not only in the case of combination processing for simplifying the panning photographing described above but also in the case of various kinds of image combination processing implemented in digital cameras proposed and sold in the market to date, for example, various kinds of image combination processing for combining images of a plurality of frames such as image combination processing for consecutively photographing images of a plurality of frames with different exposure levels and combining the images to generate an image having a larger dynamic range than an image of one frame and image combination processing for averaging and combining images of a plurality of frames to reduce random noise.